{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "7454b978-0954-4c79-87e6-b200a59d491d",
   "metadata": {},
   "source": [
    "# Understanding Image Space\n",
    "\n",
    "Let's use state-of-the art [OpenCLIP](https://github.com/mlfoundations/open_clip?tab=readme-ov-file) models to embed text or images into a multi-modal vector space."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "69abf941-d029-4a8a-b87c-b0eb43baebde",
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install superlinked==37.5.0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "69742745-d13b-4035-b7a5-6978bade5b36",
   "metadata": {},
   "outputs": [],
   "source": [
    "from io import BytesIO\n",
    "\n",
    "import requests\n",
    "import PIL\n",
    "\n",
    "from superlinked import framework as sl\n",
    "\n",
    "DATA_URL: str = \"https://storage.googleapis.com/superlinked-notebook-feature-image-embedding\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9198c3c6-9ed2-493c-8658-c0398dcf97d6",
   "metadata": {},
   "source": [
    "Let's create our dataset and load some images into it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "09278e2f-f85f-442c-9f4e-967eee8a326d",
   "metadata": {},
   "outputs": [],
   "source": [
    "def open_image_from_public_gcs(public_url: str) -> PIL.ImageFile.ImageFile:\n",
    "    # Fetch the image using the public URL\n",
    "    response = requests.get(public_url, timeout=5)\n",
    "\n",
    "    # Ensure the request was successful\n",
    "    if response.status_code == 200:\n",
    "        # Open the image with PIL using the downloaded bytes\n",
    "        downloaded_image = PIL.Image.open(BytesIO(response.content))\n",
    "        return downloaded_image\n",
    "    raise requests.RequestException(\n",
    "        f\"Failed to fetch image. Status code: {response.status_code}\",\n",
    "    )\n",
    "\n",
    "\n",
    "image_labels: list[str] = [\n",
    "    \"blue-circle\",\n",
    "    \"blue-square\",\n",
    "    \"red-circle\",\n",
    "    \"red-rectangle\",\n",
    "    \"red-circle-with-black-frame\",\n",
    "]\n",
    "\n",
    "data = [\n",
    "    {\n",
    "        \"id\": image_label,\n",
    "        \"image\": open_image_from_public_gcs(f\"{DATA_URL}/{image_label}.png\"),\n",
    "    }\n",
    "    for image_label in image_labels\n",
    "]\n",
    "\n",
    "# we will use it for querying\n",
    "blue_square_image = open_image_from_public_gcs(f\"{DATA_URL}/blue-square.png\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9aed1d85-6fba-43ce-9262-4781c67f8508",
   "metadata": {},
   "source": [
    "# Superlinked config for image search\n",
    "\n",
    "Multimodal vision transformers create the opportunity to embed the image and its caption or description together in a shared space. As a result, the embedding of the same concept will be close regardless of it being an image or a textual representation. With ImageSpace you can embed the image together with its description (or simply the image - only embedding text with ViT is currently not possible due to efficiency reasons) into this multimodal space.\n",
    "\n",
    "Use Blob SchemaFieldType for images. It accepts\n",
    "- PIL.Image in memory\n",
    "- local path or remote url to load the image from\n",
    "- imagefile as a byte array in string format"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "1d5f3257-1f43-4ab9-ab1b-7bd26be681c8",
   "metadata": {},
   "outputs": [],
   "source": [
    "class Image(sl.Schema):\n",
    "    id: sl.IdField\n",
    "    image: sl.Blob\n",
    "    description: sl.String\n",
    "\n",
    "\n",
    "image = Image()\n",
    "\n",
    "# the space is set up to aggregate the image and the description embeddings\n",
    "image_embedding_space = sl.ImageSpace(image=image.image + image.description)\n",
    "\n",
    "image_index = sl.Index(image_embedding_space)\n",
    "source: sl.InMemorySource = sl.InMemorySource(image)\n",
    "executor = sl.InMemoryExecutor(sources=[source], indices=[image_index])\n",
    "app = executor.run()\n",
    "\n",
    "simple_query = (\n",
    "    sl.Query(image_index, weights={image_embedding_space: 1.0})\n",
    "    .find(image)\n",
    "    .similar(image_embedding_space.image, sl.Param(\"image_search\"), sl.Param(\"image_weight\"))\n",
    "    .similar(\n",
    "        image_embedding_space.description,\n",
    "        sl.Param(\"text_search\"),\n",
    "        sl.Param(\"text_weight\"),\n",
    "    )\n",
    "    .select_all()\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "8a10673a-d8a1-4e4b-925d-f67a48714a29",
   "metadata": {},
   "outputs": [],
   "source": [
    "# ingest data\n",
    "source.put(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e6c10d88-da0c-4b23-8987-c9a03f101ead",
   "metadata": {},
   "source": [
    "# Run queries\n",
    "\n",
    "First on the original data, subsequently extend the dataset with descriptions.\n",
    "\n",
    "Let's search with an image of a blue square first (the same image as the ingested one, hence the 1.0 similarity)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "e34e6dcb-f51c-4636-8b90-0387dcd5c8d4",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>image</th>\n",
       "      <th>id</th>\n",
       "      <th>similarity_score</th>\n",
       "      <th>rank</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>iVBORw0KGgoAAAANSUhEUgAAA8AAAAPACAIAAAB1tIfMAA...</td>\n",
       "      <td>blue-square</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>iVBORw0KGgoAAAANSUhEUgAAAOEAAADhCAMAAAAJbSJIAA...</td>\n",
       "      <td>blue-circle</td>\n",
       "      <td>0.868173</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>iVBORw0KGgoAAAANSUhEUgAAAOEAAADhCAMAAAAJbSJIAA...</td>\n",
       "      <td>red-circle-with-black-frame</td>\n",
       "      <td>0.826916</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>iVBORw0KGgoAAAANSUhEUgAAAaQAAAEYCAYAAAATRII7AA...</td>\n",
       "      <td>red-rectangle</td>\n",
       "      <td>0.825166</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>iVBORw0KGgoAAAANSUhEUgAAAVwAAAFcBAMAAAB2OBsfAA...</td>\n",
       "      <td>red-circle</td>\n",
       "      <td>0.809378</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                               image  \\\n",
       "0  iVBORw0KGgoAAAANSUhEUgAAA8AAAAPACAIAAAB1tIfMAA...   \n",
       "1  iVBORw0KGgoAAAANSUhEUgAAAOEAAADhCAMAAAAJbSJIAA...   \n",
       "2  iVBORw0KGgoAAAANSUhEUgAAAOEAAADhCAMAAAAJbSJIAA...   \n",
       "3  iVBORw0KGgoAAAANSUhEUgAAAaQAAAEYCAYAAAATRII7AA...   \n",
       "4  iVBORw0KGgoAAAANSUhEUgAAAVwAAAFcBAMAAAB2OBsfAA...   \n",
       "\n",
       "                            id  similarity_score  rank  \n",
       "0                  blue-square          1.000000     0  \n",
       "1                  blue-circle          0.868173     1  \n",
       "2  red-circle-with-black-frame          0.826916     2  \n",
       "3                red-rectangle          0.825166     3  \n",
       "4                   red-circle          0.809378     4  "
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "result = app.query(simple_query, image_search=blue_square_image)\n",
    "sl.PandasConverter.to_pandas(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1ded1988-2a3b-45dd-b38a-567ca0826380",
   "metadata": {},
   "source": [
    "Interestingly enough, a blue circle is closer to a blue square, then a red rectangle. Also, a red circle is the furthest (understandably), but a black frame on a circle is more similar to a square than a rectangle - by a negligable margin.\n",
    "\n",
    "Now let's add descriptions to the images. Their embeddings will be aggregated with the image embeddings. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "d29a39d2-c793-4f3d-8075-9514a1dc68b0",
   "metadata": {},
   "outputs": [],
   "source": [
    "data_desc = [\n",
    "    {\n",
    "        \"id\": image_label,\n",
    "        \"image\": open_image_from_public_gcs(f\"{DATA_URL}/{image_label}.png\"),\n",
    "        \"description\": f\"This is a {' '.join(image_label.split('-'))}.\",\n",
    "    }\n",
    "    for image_label in image_labels\n",
    "]\n",
    "\n",
    "# let's try to deceive everyone by stating the red rectangle is blue and see how that affects our results\n",
    "data_desc[3][\"description\"] = \"This is a blue rectangle.\"\n",
    "\n",
    "source.put(data_desc)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "913f9e64-0e19-4628-963c-d02c357084aa",
   "metadata": {},
   "source": [
    "Continue running queries on the dataset that now contains descriptions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "3e01cb71-86cc-47c2-b303-27463316f779",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>image</th>\n",
       "      <th>description</th>\n",
       "      <th>id</th>\n",
       "      <th>similarity_score</th>\n",
       "      <th>rank</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>iVBORw0KGgoAAAANSUhEUgAAA8AAAAPACAIAAAB1tIfMAA...</td>\n",
       "      <td>This is a blue square.</td>\n",
       "      <td>blue-square</td>\n",
       "      <td>0.853752</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>iVBORw0KGgoAAAANSUhEUgAAAaQAAAEYCAYAAAATRII7AA...</td>\n",
       "      <td>This is a blue rectangle.</td>\n",
       "      <td>red-rectangle</td>\n",
       "      <td>0.753377</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>iVBORw0KGgoAAAANSUhEUgAAAOEAAADhCAMAAAAJbSJIAA...</td>\n",
       "      <td>This is a blue circle.</td>\n",
       "      <td>blue-circle</td>\n",
       "      <td>0.734472</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>iVBORw0KGgoAAAANSUhEUgAAAOEAAADhCAMAAAAJbSJIAA...</td>\n",
       "      <td>This is a red circle with black frame.</td>\n",
       "      <td>red-circle-with-black-frame</td>\n",
       "      <td>0.688376</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>iVBORw0KGgoAAAANSUhEUgAAAVwAAAFcBAMAAAB2OBsfAA...</td>\n",
       "      <td>This is a red circle.</td>\n",
       "      <td>red-circle</td>\n",
       "      <td>0.677710</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                               image  \\\n",
       "0  iVBORw0KGgoAAAANSUhEUgAAA8AAAAPACAIAAAB1tIfMAA...   \n",
       "1  iVBORw0KGgoAAAANSUhEUgAAAaQAAAEYCAYAAAATRII7AA...   \n",
       "2  iVBORw0KGgoAAAANSUhEUgAAAOEAAADhCAMAAAAJbSJIAA...   \n",
       "3  iVBORw0KGgoAAAANSUhEUgAAAOEAAADhCAMAAAAJbSJIAA...   \n",
       "4  iVBORw0KGgoAAAANSUhEUgAAAVwAAAFcBAMAAAB2OBsfAA...   \n",
       "\n",
       "                              description                           id  \\\n",
       "0                  This is a blue square.                  blue-square   \n",
       "1               This is a blue rectangle.                red-rectangle   \n",
       "2                  This is a blue circle.                  blue-circle   \n",
       "3  This is a red circle with black frame.  red-circle-with-black-frame   \n",
       "4                   This is a red circle.                   red-circle   \n",
       "\n",
       "   similarity_score  rank  \n",
       "0          0.853752     0  \n",
       "1          0.753377     1  \n",
       "2          0.734472     2  \n",
       "3          0.688376     3  \n",
       "4          0.677710     4  "
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "result = app.query(simple_query, image_search=blue_square_image)\n",
    "sl.PandasConverter.to_pandas(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f22cc9c9-2242-4dc3-a825-e737b5831377",
   "metadata": {},
   "source": [
    "Our trick worked, now the red rectangle (labeled blue) is the closest item to a blue square (taking over the blue circle, that is actually blue in description and image as well.)\n",
    "\n",
    "Now let's try searching with some text in the vectorspace of the Vision Transformer - showing that textual queries can be used, too."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "fc246016-ec52-42bf-b145-8a13123835ef",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>image</th>\n",
       "      <th>description</th>\n",
       "      <th>id</th>\n",
       "      <th>similarity_score</th>\n",
       "      <th>rank</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>iVBORw0KGgoAAAANSUhEUgAAAOEAAADhCAMAAAAJbSJIAA...</td>\n",
       "      <td>This is a red circle with black frame.</td>\n",
       "      <td>red-circle-with-black-frame</td>\n",
       "      <td>0.746052</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>iVBORw0KGgoAAAANSUhEUgAAAVwAAAFcBAMAAAB2OBsfAA...</td>\n",
       "      <td>This is a red circle.</td>\n",
       "      <td>red-circle</td>\n",
       "      <td>0.697133</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>iVBORw0KGgoAAAANSUhEUgAAAOEAAADhCAMAAAAJbSJIAA...</td>\n",
       "      <td>This is a blue circle.</td>\n",
       "      <td>blue-circle</td>\n",
       "      <td>0.648808</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>iVBORw0KGgoAAAANSUhEUgAAAaQAAAEYCAYAAAATRII7AA...</td>\n",
       "      <td>This is a blue rectangle.</td>\n",
       "      <td>red-rectangle</td>\n",
       "      <td>0.596682</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>iVBORw0KGgoAAAANSUhEUgAAA8AAAAPACAIAAAB1tIfMAA...</td>\n",
       "      <td>This is a blue square.</td>\n",
       "      <td>blue-square</td>\n",
       "      <td>0.563707</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                               image  \\\n",
       "0  iVBORw0KGgoAAAANSUhEUgAAAOEAAADhCAMAAAAJbSJIAA...   \n",
       "1  iVBORw0KGgoAAAANSUhEUgAAAVwAAAFcBAMAAAB2OBsfAA...   \n",
       "2  iVBORw0KGgoAAAANSUhEUgAAAOEAAADhCAMAAAAJbSJIAA...   \n",
       "3  iVBORw0KGgoAAAANSUhEUgAAAaQAAAEYCAYAAAATRII7AA...   \n",
       "4  iVBORw0KGgoAAAANSUhEUgAAA8AAAAPACAIAAAB1tIfMAA...   \n",
       "\n",
       "                              description                           id  \\\n",
       "0  This is a red circle with black frame.  red-circle-with-black-frame   \n",
       "1                   This is a red circle.                   red-circle   \n",
       "2                  This is a blue circle.                  blue-circle   \n",
       "3               This is a blue rectangle.                red-rectangle   \n",
       "4                  This is a blue square.                  blue-square   \n",
       "\n",
       "   similarity_score  rank  \n",
       "0          0.746052     0  \n",
       "1          0.697133     1  \n",
       "2          0.648808     2  \n",
       "3          0.596682     3  \n",
       "4          0.563707     4  "
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "result = app.query(simple_query, text_search=\"black frame around a red circle\")\n",
    "sl.PandasConverter.to_pandas(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "57cdde7d-feab-412d-b2a3-e765c768e124",
   "metadata": {},
   "source": [
    "Utilising the 2 similar clauses in the query, we can search with text and image at the same time..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "870356f4-c19f-4cdc-8cff-556ff50ebf33",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>image</th>\n",
       "      <th>description</th>\n",
       "      <th>id</th>\n",
       "      <th>similarity_score</th>\n",
       "      <th>rank</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>iVBORw0KGgoAAAANSUhEUgAAA8AAAAPACAIAAAB1tIfMAA...</td>\n",
       "      <td>This is a blue square.</td>\n",
       "      <td>blue-square</td>\n",
       "      <td>0.898397</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>iVBORw0KGgoAAAANSUhEUgAAAVwAAAFcBAMAAAB2OBsfAA...</td>\n",
       "      <td>This is a red circle.</td>\n",
       "      <td>red-circle</td>\n",
       "      <td>0.896603</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>iVBORw0KGgoAAAANSUhEUgAAAOEAAADhCAMAAAAJbSJIAA...</td>\n",
       "      <td>This is a blue circle.</td>\n",
       "      <td>blue-circle</td>\n",
       "      <td>0.891934</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>iVBORw0KGgoAAAANSUhEUgAAAOEAAADhCAMAAAAJbSJIAA...</td>\n",
       "      <td>This is a red circle with black frame.</td>\n",
       "      <td>red-circle-with-black-frame</td>\n",
       "      <td>0.891678</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>iVBORw0KGgoAAAANSUhEUgAAAaQAAAEYCAYAAAATRII7AA...</td>\n",
       "      <td>This is a blue rectangle.</td>\n",
       "      <td>red-rectangle</td>\n",
       "      <td>0.858931</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                               image  \\\n",
       "0  iVBORw0KGgoAAAANSUhEUgAAA8AAAAPACAIAAAB1tIfMAA...   \n",
       "1  iVBORw0KGgoAAAANSUhEUgAAAVwAAAFcBAMAAAB2OBsfAA...   \n",
       "2  iVBORw0KGgoAAAANSUhEUgAAAOEAAADhCAMAAAAJbSJIAA...   \n",
       "3  iVBORw0KGgoAAAANSUhEUgAAAOEAAADhCAMAAAAJbSJIAA...   \n",
       "4  iVBORw0KGgoAAAANSUhEUgAAAaQAAAEYCAYAAAATRII7AA...   \n",
       "\n",
       "                              description                           id  \\\n",
       "0                  This is a blue square.                  blue-square   \n",
       "1                   This is a red circle.                   red-circle   \n",
       "2                  This is a blue circle.                  blue-circle   \n",
       "3  This is a red circle with black frame.  red-circle-with-black-frame   \n",
       "4               This is a blue rectangle.                red-rectangle   \n",
       "\n",
       "   similarity_score  rank  \n",
       "0          0.898397     0  \n",
       "1          0.896603     1  \n",
       "2          0.891934     2  \n",
       "3          0.891678     3  \n",
       "4          0.858931     4  "
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "result = app.query(simple_query, image_search=blue_square_image, text_search=\"red circle\")\n",
    "sl.PandasConverter.to_pandas(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ed974f3c-5f8a-4339-8279-15a617023170",
   "metadata": {},
   "source": [
    "... and even set different weights according to the importance of the respective search terms."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "1b16624a-722a-49e8-be10-a43c4f27c0a0",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>image</th>\n",
       "      <th>description</th>\n",
       "      <th>id</th>\n",
       "      <th>similarity_score</th>\n",
       "      <th>rank</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>iVBORw0KGgoAAAANSUhEUgAAAVwAAAFcBAMAAAB2OBsfAA...</td>\n",
       "      <td>This is a red circle.</td>\n",
       "      <td>red-circle</td>\n",
       "      <td>0.775167</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>iVBORw0KGgoAAAANSUhEUgAAAOEAAADhCAMAAAAJbSJIAA...</td>\n",
       "      <td>This is a red circle with black frame.</td>\n",
       "      <td>red-circle-with-black-frame</td>\n",
       "      <td>0.758307</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>iVBORw0KGgoAAAANSUhEUgAAAOEAAADhCAMAAAAJbSJIAA...</td>\n",
       "      <td>This is a blue circle.</td>\n",
       "      <td>blue-circle</td>\n",
       "      <td>0.718329</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>iVBORw0KGgoAAAANSUhEUgAAAaQAAAEYCAYAAAATRII7AA...</td>\n",
       "      <td>This is a blue rectangle.</td>\n",
       "      <td>red-rectangle</td>\n",
       "      <td>0.651391</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>iVBORw0KGgoAAAANSUhEUgAAA8AAAAPACAIAAAB1tIfMAA...</td>\n",
       "      <td>This is a blue square.</td>\n",
       "      <td>blue-square</td>\n",
       "      <td>0.623732</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                               image  \\\n",
       "0  iVBORw0KGgoAAAANSUhEUgAAAVwAAAFcBAMAAAB2OBsfAA...   \n",
       "1  iVBORw0KGgoAAAANSUhEUgAAAOEAAADhCAMAAAAJbSJIAA...   \n",
       "2  iVBORw0KGgoAAAANSUhEUgAAAOEAAADhCAMAAAAJbSJIAA...   \n",
       "3  iVBORw0KGgoAAAANSUhEUgAAAaQAAAEYCAYAAAATRII7AA...   \n",
       "4  iVBORw0KGgoAAAANSUhEUgAAA8AAAAPACAIAAAB1tIfMAA...   \n",
       "\n",
       "                              description                           id  \\\n",
       "0                   This is a red circle.                   red-circle   \n",
       "1  This is a red circle with black frame.  red-circle-with-black-frame   \n",
       "2                  This is a blue circle.                  blue-circle   \n",
       "3               This is a blue rectangle.                red-rectangle   \n",
       "4                  This is a blue square.                  blue-square   \n",
       "\n",
       "   similarity_score  rank  \n",
       "0          0.775167     0  \n",
       "1          0.758307     1  \n",
       "2          0.718329     2  \n",
       "3          0.651391     3  \n",
       "4          0.623732     4  "
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "result = app.query(\n",
    "    simple_query,\n",
    "    image_search=blue_square_image,\n",
    "    image_weight=0.1,\n",
    "    text_search=\"red circle\",\n",
    "    text_weight=1.0,\n",
    ")\n",
    "sl.PandasConverter.to_pandas(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "12e69bdf-17ca-4f96-88d0-d1c1952c5189",
   "metadata": {},
   "source": [
    "Notice how the weight change moves everything red and circular upwards, while blue and rectangulars move downwards in the result ranking."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "superlinked-py3.11",
   "language": "python",
   "name": "superlinked-py3.11"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
